What should a bit-tech iPhone 4 review look like?

How would we test iPhone 4?

So, it’s clearly a fascinating area and one I think we should cover. The question is how? As mentioned previously, there’s a lot of smartphone coverage out there, and a lot of it is very good – certainly if you’re a mainstream user, or want a quick preview of a new device as soon as it’s available. But that’s not what we’re here for. We’re about power users, enthusiasts, geeks and tinkerers.

The clear gap, it seems to me, is benchmarking. Current online coverage is great at giving subjective impressions – and these are important – but few smartphone reviews include any kind of benchmarks; if they do, they usually extend only to a battery life test and perhaps a run through of SunSpider, a benchmark developed by Apple's WebKit team which tests a browser's JavaScript performance.

Certainly for most bit-tech readers, I'd imagine if you had the choice between reading a review and playing with the kit yourself, you'd chose the latter – and decent data from benchmarks is a great way of bringing a reader closer to a device or piece of hardware (though it’s not the only way, of course).


By benchmarking, I don’t just mean graphs and numbers (although we do a love a good graph). What I really mean is an objective assessment of a device, through testing – features, how they are to use, and of course, the cost versus the competition. With smartphones, surely total cost of ownership should be a consideration? None of the early iPhone 4 reviews touched on cost in any meaningful way, yet it’s a key way in which Android and Windows Phone 7, with their jostle of manufacturers and networks, intend to compete against the iPhone. The iPhone is certainly powerful, but on a 24 month contract, a 16GB iPhone 4 will cost at least £750 in the UK, the price pushed up by the fact you can only get a free iPhone on very expensive monthly plans.

Then there’s speed. Three or four years ago, smartphones were limited in what they could do and a benchmarking approach solely limited to battery life was appropriate. The world is very different now. Companies market smartphones based on their performance. Hardware and software have both advanced, so you're able to do a lot more with these devices. You can install third party software, shoot and edit HD video, open multiple browser pages, run games and multi-task.

Benchmarking is important to us because higher performance computers are generally better to use. Whether it's games running more smoothly, tough tasks such as video encoding completing more quickly, or simply a computer that doesn't frustrate you with long wait times. Performance matters to this stuff; I notice the difference between my iPhone 3G and my fiancee's 3GS - she can keep eight browser windows open without the phone slowing down. Plants versus Zombies loads noticeably quicker. The phone as a whole feels more responsive. I'm now out of contract and looking for a new phone - and speed is a key issue for me, and it's one I'd want to see covered in reviews in a way that feels substantial, meaningful and scientific.

It’s an area where a lot of smartphone coverage is unsatisfactory, to my (admittedly geeky) mind – endless reviews of the newest phones praise each one as “super fast” or “snappy”. Consider this paragraph from Engadget’s otherwise excellent iPhone 4 review:

“While the company [Apple] hasn't yet said what the clock speed of the processor is, we're guessing it's something below the 1GHz touted for its tablet cousin. The phone is definitely snappier than the 3GS, so we're not about to volley complaints just yet -- in particular, graphics seemed to render faster, and overall responsiveness was slightly higher, though admittedly, it wasn't blowing the doors off the joint. It's certainly faster, but the 3GS wasn't hurting on speed to our eyes, so it's not as wildly noticeable a leap as the 3G to the 3GS.”

It’s as vague and woolly as a post-match interview with a football playing woolly Mammoth. Graphics render faster? What sort of graphics? Are we talking about 3D or just the menus? The phone is always drawing graphics, after all. It’s just a giant screen. I should point out that aside from this, the Engadget review is admirably precise and detailed – if anything, the reason they’re flailing here is because they know performance is important, they just don’t know how to talk about it, and in that respect, we’re with them on both counts.

Smartphone performance is going to be increasingly relevant as we see a variety of devices running the same software, but from different manufacturers. If you're shopping for an Android phone, or a Windows 7 Phone, the software will be largely the same, but HTC, Motorola, LG, Samsung, Sony will all be using different hardware. It's not just the external design that's different either. Thanks to ARMs licensing approach, partners are free to build their own CPU or buy from a variety of third-party vendors.

There are at least three companies (TI, Qualcomm and Samsung) who make 1GHz ARM Coretex A8 chips next-generation smartphones will use. As with fully fledged computers, the phones differ in terms of how much system memory they're available with too, and the size and resolution of the screen. There are enough variables that performance testing will be helpful to sort through. Once Intel get into the game, it’s going to get even more interesting.